The Parallel Evolution of Search Engines and Digital Libraries: Their Convergence to the Mega-Portal

The information era has brought with it the well-known problem of 'Information Explosion'. There are many and varied Search Engines (SEs) on the Internet but it is still hard to locate and concentrate only on materials relevant to a specific task. Digital Libraries (DLs) could better provide such services on the Web. However, DLs have been much less researched and implemented than SEs. In any case, there is a real need to formulate a methodology for the understanding of both these types of Web data repositories, and especially of digital libraries. In this paper we classify both SEs and DLs using similar criteria on both a functional scale and a generational time-line. In particular, we introduce the idea of the third generation Harvested DL (HDL) and its resulting DL harvesting model. The DL harvesting model was implemented in the ‘Katsir’ DL prototype system. Until recently, SEs and DLs were seen as completely separate paradigms dealing with building Web data repositories. By comparing them here and analyzing their characteristics, we discover that they actually share much in common. Our main objective here is to better understand what SEs and DLs really are, and what was the parallel evolution of SEs and DLs in the existing three generations. Finally, we note the expected incorporation of intelligent techniques and knowledge management in fourth generation SEs and DLs and the expected convergence of their interfaces and structures in the fifth generation, coined the MegaPortal. This concept generalizes all the features and characteristics expected to be found and implemented in the future Web Repository.